Various database management system functions operate behind the scenes, preserving the value of the database and its contents. Two such time dependent, sequential, database management system functions are recovery logging and data replication.
In recovery logging the system maintains a log or journal to keep track of all operations that affect the values of database items. This log or journal records the history of database changes in a secure way and contains details of all updates to the database. The log may include “before” and “after” images of the state of an updated object. Thus, if it becomes necessary to undo some particular update, the system can use the corresponding log entry to restore the updated object to the previous value. The log or journal may be used to permit recovery from application or system errors or failures, and recover the consistency of the database right up to the point in time where an error occurred.
The log files are created automatically by the database management system when the database is created. Typically, the log consists of two portions, an “online” or “active” portion and an “archive” or “offline” portion. The “online” portion is the portion used during normal system operation to record details of the updates as they are performed, and is normally held on a local disk. When the online portion becomes full, or more frequently, its contents are transferred to the offline portion, which, because it is always processed sequentially, can be held on disk or in a tertiary storage device.
If the system crashes, it is possible to recover to a transaction consistent database state by examining the log and using various recovery algorithms, such as the one described by C. Mohan, D. Haderle, B. Lindsay, H. Pirahesh, and P. Schwarz, in “ARIES: A Transaction Recovery Method Supporting Fine Granularity Locking and Partial Rollbacks Using Write Ahead Logging” TODS 17(1): pp. 94-152 (1992). With the recovery log, it is possible to undo WRITE operations by tracing backward through the log and resetting all items changed by a WRITE operation of an identified TransactionID to their Old_Value. Likewise, redoing may be needed if all of the updates are recorded in the log, but a failure has occurred before all of the New_Values have been written (permanently) to the database. Redoing is accomplished by tracing forward through the log and setting all of the items changed by a WRITE operation of TransactionID to their New_Values.
Replication is a process of maintaining a defined set of database content, including database entries, and files, in more than one database. Replication involves copying (capturing) designated changes from one database (a source) and applying designated changes to another database (a target). The source and target can be in the same or different logical servers that are on the same machine or on different machines in a distributed network or different networks.
Data replication includes a “capture” process at the source database, an “apply” process at the target database, and “clean up” processes at both databases, and uses a plurality of co-ordination, control, communications, timing, and logging functions and applications. Data replication applications, such as IBM DB2 DataPropagator, consist of three main components: administration interfaces, change-capture mechanisms, and change-apply mechanisms. A user uses the administration interfaces to create a replication environment, including control tables, that designate data to be replicated and control the replication methods and processes. The control tables store user replication criteria and monitor data replication progress. After the user sets up a user replication environment, the user uses a change-capture mechanism to capture changes that may not be contemporaneous and that occur in the source database and store them temporarily in tables. Finally, the user uses the apply program to apply the captured entries to target databases, or to copy data directly from the source database to the target databases.
The capture and apply replication components use a variety of control tables, logs, timestamps, buffers, and search engines to communicate with each other and to manage replication requests, such as defining and managing replication sources and targets, capturing changes, replicating changes, tracking how many changes are replicated and how many remain to be done, and applying changes to the target.
The replication components reside on one or more logical servers. As used herein, logical servers refer to databases, not to hardware servers in the client/server sense. For the IBM OS/390 operating system, logical servers are equivalent to subsystems or data-sharing groups (that is, the domain of a single database catalog). There are three types of logical servers in replication: a source server, a target server, and a control server.
The source server contains the change-capture mechanism, the source tables that a user wants to replicate, and the control tables for the capture program. The target server contains the target tables. The control server contains the control tables for the apply program.
The apply program can reside on any of the logical servers in the network. Each apply program is associated with one control server, which a user specifies when the user starts the apply program. Multiple apply programs can share a control server.
A DBMS data replication solution offers various mechanisms for capturing data, including analysis of the contents of the recovery log. For example, when the source is an IBM DB2 table, the capture program is used to capture changes that are made to the source. The capture program interrogates the contents of the database recovery log to deduce changes made to the source database and to record them temporarily for transfer to the apply program at the target database. Typically the capture program runs at the machine hosting the source database server. While the capture program may run continuously, a user can stop capture while running utilities or modifying replication sources. Tasks that a user can perform with the capture program include starting the capture program, scheduling the capture program, stopping the capture program, suspending the capture program temporarily, resuming the capture program, reinitializing the capture program, and pruning tables that temporarily store change data. Additionally, tasks associated with starting up and shutting down replication from specific servers' data tables are initiated by user command, for example, from administrative tools and interfaces.
Once the apply program has been started, the apply program reads data directly from source tables or views to initially populate the target objects in the target database table. If a user wants changes captured, the apply program reads the changed data that was previously captured and stored temporarily in staging tables, and applies the changes to target tables.
Tasks that a user can perform with the Apply program include starting the apply program, running exit routines (such as ASNLOAD to call an IBM or other utility), scheduling the apply program, and stopping the apply program.
The replication components are independent of each other; they rely on information that is stored in control tables to communicate with each other. The capture and apply programs update control tables to indicate the progress of replication and to coordinate the processing of changes.
Each time that the apply program copies data to the target database, the contents of the target database reflect the changes that were made to the source database. The apply program works by writing the updates accumulated since the apply program last ran. The Apply program keeps track of the latest update that it makes to each target.
To be noted is that the capture program will not capture any information until the apply program signals it to do so, and the apply program will not signal the capture program to start capturing changes until a user defines a replication source and associated subscription sets.
Replication may be synchronous or asynchronous. Synchronous replication delivers updates continually. When a change is made to the source data, it is immediately applied to the target database. A change is committed to the source database at the same time as the change is committed to the target database. If for some reason the change cannot be replicated to the target database, the change is not made to the source database. This type of replication is also called real-time replication. If a user application requires synchronous updates, the user application updates all of the tables in a single, distributed transaction.
Asynchronous replication delivers updates asynchronously to the process making changes to the source data. When a change is made to the source data, it is stored temporarily for a preset interval and forwarded to the target at a later time. The interval can be a measure of time (seconds, minutes, hours) or can represent a prescribed event (midnight, or some other time of day) or it can be best effort, as soon as possible. If changes cannot be made to a target database (for example, if the target database is down or the network is down), the changes are stored and applied later, in the order in which they were made to the source. Asynchronous replication provides certain identifiable benefits over synchronous replication: better use of network resources, less database contention, and the opportunity to enhance data before it reaches the target database.
With asynchronous replication, changes made to the source are not processed immediately. Rather, they are processed later when the replication system encounters the changes in the recovery log.
An important requirement of data replication systems is the need to synchronize data replication events with actions taken by the users and administrators. However, when the data capture process is, fundamentally, driven by the contents of the recovery log, the data replication process or method operates asynchronously to when the events are recorded in the log. Thusly, we perceive the need for a user or administrator to have a means (and mechanism) to direct the replication system to take specified actions (only) when the replication system is processing recovery log entries corresponding to the time when the user or administrator requested the action. For example, if a user wishes to stop replication of selected source data at, say, 10 AM, the user can take some action at 10 AM. But, at 10 AM, the replication system will not yet have processed the entries made to the recovery log before 10 AM. Thus, a direct (synchronous) communication between the user and the replication system at 10 AM can not be immediately used to suppress replication of changes that happen after 10 AM.
Thus, a clear need exists to provide a system, a process, and a program product for synchronizing an event with a position in the event sequence embodied in the recovery log.